Skip to content

[None][fix] Allow MiniMax-M3's to use separate draft KV cache in disagg - #17341

Merged
zheyuf merged 1 commit into
NVIDIA:feat/m3_with_msafrom
zheyuf:fix/m3-eagle3-disagg-separate-draft-kv-cache
Aug 6, 2026
Merged

[None][fix] Allow MiniMax-M3's to use separate draft KV cache in disagg#17341
zheyuf merged 1 commit into
NVIDIA:feat/m3_with_msafrom
zheyuf:fix/m3-eagle3-disagg-separate-draft-kv-cache

Conversation

@zheyuf

@zheyuf zheyuf commented Aug 6, 2026

Copy link
Copy Markdown
Collaborator

Pipe clean disagg + eagle. This PR makes disagg+eagle runnable. TODO in future PR: support disagg + seperate kv cache to restore AL.

PR Checklist

  • PR title follows the [JIRA/NVBUG/None][type] format
  • Commit is signed off (DCO)
  • yapf/isort clean on touched files
  • CI (/bot run)

@zheyuf
zheyuf force-pushed the fix/m3-eagle3-disagg-separate-draft-kv-cache branch from 65db48f to bb6a4c7 Compare August 6, 2026 08:58
…he in disaggregated serving

The nvbug 5807902 WAR disables the separate draft KV cache manager
whenever a cache transceiver is configured. MiniMax-M3 cannot take the
shared-manager path that WAR forces: its cache manager declares
supports_shared_draft_layers=False, and the drafter then inherits the
target's tokens_per_block=128 pages, which miss the SM10x Eagle context
cubins (the unfused-MHA fallback requests a 6.17 TiB workspace on a
real 32K-token warmup) and hit the known tokens_per_block=128
trtllm-gen generation-kernel IMA. Both context and generation workers
crashed during startup on every disaggregated Eagle3 attempt.

Exempt MiniMax-M3 from the WAR so both worker roles keep the designed
tokens_per_block=32 separate draft manager (symmetry is required for a
consistent target pool layout across the KV transfer).

Validated on Lyris GB300 (2xCTX TP2 + GEN TP4/ADP, NIXL): startup
completes end to end, and the test_nvfp4_eagle3 chat-GSM8K acceptance
workload measures AL 3.330 disagg vs 3.474 aggregated on the same
build (drafter card reference 3.518). The remaining gap is the
transceiver not transferring draft-layer KV, tracked separately.

Signed-off-by: Zheyu Fu <zheyuf@nvidia.com>
@zheyuf
zheyuf force-pushed the fix/m3-eagle3-disagg-separate-draft-kv-cache branch from bb6a4c7 to 719dee1 Compare August 6, 2026 09:02
@zheyuf zheyuf changed the title [https://nvbugs/5807902][fix] Keep MiniMax-M3's separate draft KV cache in disaggregated serving [None][fix] Allow MiniMax-M3's to use separate draft KV cache in disagg Aug 6, 2026
@zheyuf
zheyuf marked this pull request as ready for review August 6, 2026 09:05
@zheyuf
zheyuf requested a review from a team as a code owner August 6, 2026 09:05
@zheyuf
zheyuf requested review from allisonlim-nv and mikeiovine and removed request for a team, allisonlim-nv and mikeiovine August 6, 2026 09:05
@zheyuf

zheyuf commented Aug 6, 2026

Copy link
Copy Markdown
Collaborator Author

/bot run

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #64291 [ run ] triggered by Bot. Commit: 719dee1 Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #64291 [ run ] completed with state SUCCESS. Commit: 719dee1
/LLM/main/L0_MergeRequest_PR pipeline #52191 completed with status: 'SUCCESS'

CI Report

Link to invocation

@zheyuf
zheyuf requested review from brb-nv and peihu-nv August 6, 2026 16:27
@brb-nv

brb-nv commented Aug 6, 2026

Copy link
Copy Markdown
Collaborator

Let's add a disagg accuracy test with Eagle to protect the functionality on side-branch, Zheyu?
If you think time is a concern, we can replace an agg + eagle3 parameterization.

Comment thread tensorrt_llm/_torch/pyexecutor/py_executor_creator.py
@zheyuf
zheyuf requested a review from brb-nv August 6, 2026 17:09

@brb-nv brb-nv left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@zheyuf
zheyuf merged commit 1368564 into NVIDIA:feat/m3_with_msa Aug 6, 2026
16 of 20 checks passed
@zheyuf

zheyuf commented Aug 7, 2026

Copy link
Copy Markdown
Collaborator Author

Let's add a disagg accuracy test with Eagle to protect the functionality on side-branch, Zheyu? If you think time is a concern, we can replace an agg + eagle3 parameterization.

Yes I will add the disagg accuracy test after the disagg+eagle get fixed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants